Support colocated joins when a partition holds no segments - #19166

Why

A colocated join fails today when a partition of one of its tables holds no segments:

Failed to find any segment for table: myTable_OFFLINE, partition: 3

This is easy to hit and hard to avoid:

• A partition key with skew leaves some partitions with no rows.
• Retention removes every segment of a partition.
• A stream partition exists but has never committed a segment.
• One logical partition space is split across several tables, so each table populates a subset.
• A table declares more partitions than it currently uses, to leave room to grow.

The assignment cannot simply skip the empty partition. Worker ids come from a counter over the partitions that hold data, so skipping one shifts every later partition down a slot. Two tables that each drop a different empty partition then end up with equal worker counts, get wired one-to-one, and join mismatched partitions. That loses rows and reports no error, which is why the assignment refuses to continue at all.

What changes

• The stages tied together by direct exchanges form a group, and the group shares one ordered list of partition classes.
• A class leaves the list only when no member of the group holds data in it.
• A class the group keeps, but one member holds no data for, gets a worker with no segments. That worker sits on a server borrowed from a member that does hold the class, so the exchange stays in process.
• Both sides of every direct exchange assert that they agree on the list. A disagreement fails at plan time instead of returning wrong rows.
• The broker now publishes the partitions whose only segments are new and have no online replica. Those hold data that no server can serve as a whole, so they keep failing.

A side effect is less fan-out. A table that declares 8 partitions and populates 3 runs 3 workers, so the broker sends fewer requests and waits on fewer servers.

What now fails on purpose

• A hybrid table with segments whose partition metadata does not match the table config now fails planning. It used to answer and drop those segments in silence. The most common cause is a partition config added after some segments were built.
• A table spread over more than one cluster reports no partition info, so a partition-aware plan is not attempted on a partial view. No server holds another cluster's segments, so one cluster's view would make a partition served elsewhere look empty.

Broker pruning comes next

Broker pruning is off for a colocated join today, and this change is what makes it possible.

• Pruning drops partitions that hold no matching segment. That is the same operation as dropping a class that holds no data.
• The group already shares one class list, so a pruned class can leave the list the same way an empty one does, and both sides stay in step.
• The gain is fewer servers per query, which protects tail latency on large clusters.

A follow-up PR adds it.

Testing

• 19 files. Planner, runtime and broker suites all pass.
• An end-to-end test joins two tables that declare 8 partitions and populate 3 and 4 of them. It asserts the rows, the worker count per leaf, the segments each leaf reads, and that each send goes to one receiver with the same worker id. That last check proves the plan did not fall back to a shuffle.
• One test drives the broker metadata directly and proves the new signal does not depend on map iteration order.

Three things left out on purpose

Aggregation merge identity is not covered here. A leaf that scans nothing emits one identity row for an aggregation with no GROUP BY. That predates this change: a worker whose segments are all pruned on the server already does the same. This change raises how many such rows reach the merge without introducing the dependency. The one aggregation that is not a true merge identity, PERCENTILEKLL, fails only when every worker is empty, which this change does not newly reach. It is tracked separately.

A worker with no segments is charged against the query thread estimate like any other worker. It is dispatched and does run a leaf operator. The estimate over-counts by two threads per such worker. That is conservative, and it only affects colocated joins over a partition space that is largely unpopulated. Those queries failed outright before, so there is no earlier estimate to compare against.

A partitioned table outside a colocated join still fails on an empty partition. A shuffled send puts no constraint on its worker ids, so the leaf joins no group and gets no class list. The error is the same one this change removes for colocated joins, and it predates this change.

Labels

Applied: bug, backward-incompat, release-notes, multi-stage, query